Synaptic Information Storage Capacity Measured With Information Theory.

Variation in the strength of synapses can be quantified by measuring the anatomical properties of synapses. Quantifying precision of synaptic plasticity is fundamental to understanding information storage and retrieval in neural circuits. Synapses from the same axon onto the same dendrite have a common history of coactivation, making them ideal candidates for determining the precision of synaptic plasticity based on the similarity of their physical dimensions. Here, the precision and amount of information stored in synapse dimensions were quantified with Shannon information theory, expanding prior analysis that used signal detection theory (Bartol et al., 2015). The two methods were compared using dendritic spine head volumes in the middle of the stratum radiatum of hippocampal area CA1 as well-defined measures of synaptic strength. Information theory delineated the number of distinguishable synaptic strengths based on nonoverlapping bins of dendritic spine head volumes. Shannon entropy was applied to measure synaptic information storage capacity (SISC) and resulted in a lower bound of 4.1 bits and upper bound of 4.59 bits of information based on 24 distinguishable sizes. We further compared the distribution of distinguishable sizes and a uniform distribution using Kullback-Leibler divergence and discovered that there was a nearly uniform distribution of spine head volumes across the sizes, suggesting optimal use of the distinguishable values. Thus, SISC provides a new analytical measure that can be generalized to probe synaptic strengths and capacity for plasticity in different brain regions of different species and among animals raised in different conditions or during learning. How brain diseases and disorders affect the precision of synaptic plasticity can also be probed.

Life and fate of Ervin Bauer (1890-1938), the eminent scholar and foundational theoretical biologist.

Ervin Bauer (1890-1938) was the first to build a general molecular-based biological theory. He defined the basic principles of theoretical biology from a thermodynamic perspective, focusing on the capacity of biological systems to produce and support the state of sustainable non-equilibrium. His central work "Theoretical Biology" (1935) was written long before modern advances in molecular biology, genetics, and information theory. Ervin Bauer and his wife Stefánia were executed in Stalin's Great Terror. This paper presents a brief introduction to Ervin Bauer's life and includes his short biography.

Multi-view graph pooling with coarsened graph disentanglement.

Multi-view graph pooling utilizes information from multiple perspectives to generate a coarsened graph, exhibiting superior performance in graph-level tasks. However, existing methods mainly focus on the types of multi-view information to improve graph pooling operations, lacking explicit control over the pooling process and theoretical analysis of the relationships between views. In this paper, we rethink the current paradigm of multi-view graph pooling from an information theory perspective, subsequently introducing GDMGP, an innovative method for multi-view graph pooling derived from the principles of graph disentanglement. This approach effectively simplifies the original graph into a more structured, disentangled coarsened graph, enhancing the clarity and utility of the graph representation. Our approach begins with the design of a novel view mapper that dynamically integrates the node and topology information of the original graph. This integration enhances its information sufficiency. Next, we introduce a view fusion mechanism based on conditional entropy to accurately regulate the task-relevant information in the views, aiming to minimize information loss in the pooling process. Finally, to further enhance the expressiveness of the coarsened graph, we disentangle the fused view into task-relevant and task-irrelevant subgraphs through mutual information minimization, retaining the task-relevant subgraph for downstream tasks. We theoretically demonstrate that the performance of the coarsened graph generated by our GDMGP is superior to that of any single input view. The effectiveness of GDMGP is further validated by experimental results on seven public datasets.

Consciousness transitions during epilepsy seizures through the lens of integrated information theory.

Consciousness is one of the most complex aspects of human experience. Studying the mechanisms involved in the transitions among different levels of consciousness remains as one of the greatest challenges in neuroscience. In this study we use a measure of integrated information (ΦAR) to evaluate dynamic changes during consciousness transitions. We applied the measure to intracranial electroencephalography (SEEG) recordings collected from 6 patients that suffer from refractory epilepsy, taking into account inter-ictal, pre-ictal and ictal periods. We analyzed the dynamical evolution of ΦAR in groups of electrode contacts outside the epileptogenic region and compared it with the Consciousness Seizure Scale (CCS). We show that changes on ΦAR are significantly correlated with changes in the reported states of consciousness.

Towards generalizable Graph Contrastive Learning: An information theory perspective.

Graph Contrastive Learning (GCL) is increasingly employed in graph representation learning with the primary aim of learning node/graph representations from a predefined pretext task that can generalize to various downstream tasks. Meanwhile, the transition from a specific pretext task to diverse and unpredictable downstream tasks poses a significant challenge for GCL's generalization ability. Most existing GCL approaches maximize mutual information between two views derived from the original graph, either randomly or heuristically. However, the generalization ability of GCL and its theoretical principles are still less studied. In this paper, we introduce a novel metric GCL-GE, to quantify the generalization gap between predefined pretext and agnostic downstream tasks. Given the inherent intractability of GCL-GE, we leverage concepts from information theory to derive a mutual information upper bound that is independent of the downstream tasks, thus enabling the metric's optimization despite the variability in downstream tasks. Based on the theoretical insight, we propose InfoAdv, a GCL framework to directly enhance generalization by jointly optimizing GCL-GE and InfoMax. Extensive experiments validate the capability of InfoAdv to enhance performance across a wide variety of downstream tasks, demonstrating its effectiveness in improving the generalizability of GCL.

The Missing Person problem through the lens of information theory.

Missing person cases typically require a genetic kinship test to determine the relationship between an unidentified individual and the relatives of the missing person. When not enough genetic evidence has been collected the lack of statistical power of these tests might lead to unreliable results. This is particularly true when just a few distant relatives are available for genotyping. In this contribution, we considered a Bayesian network approach for kinship testing and proposed several information theoretic metrics in order to quantitatively evaluate the information content of pedigrees. We show how these statistics are related to the widely used likelihood ratio values and could be employed to efficiently prioritize family members in order to optimize the statistical power in missing person problems. Our methodology seamlessly integrates with Bayesian modeling approaches, like the GENis platform that we have recently developed for high-throughput missing person identification tasks. Furthermore, our approach can also be easily incorporated into Elston-Stewart forensic frameworks. To facilitate the application of our methodology, we have developed the forensIT package, freely available on CRAN repository, which implements all the methodologies described in our manuscript.

Modifying the false discovery rate procedure based on the information theory under arbitrary correlation structure and its performance in high-dimensional genomic data.

BACKGROUND: Controlling the False Discovery Rate (FDR) in Multiple Comparison Procedures (MCPs) has widespread applications in many scientific fields. Previous studies show that the correlation structure between test statistics increases the variance and bias of FDR. The objective of this study is to modify the effect of correlation in MCPs based on the information theory. We proposed three modified procedures (M1, M2, and M3) under strong, moderate, and mild assumptions based on the conditional Fisher Information of the consecutive sorted test statistics for controlling the false discovery rate under arbitrary correlation structure. The performance of the proposed procedures was compared with the Benjamini-Hochberg (BH) and Benjamini-Yekutieli (BY) procedures in simulation study and real high-dimensional data of colorectal cancer gene expressions. In the simulation study, we generated 1000 differential multivariate Gaussian features with different levels of the correlation structure and screened the significance features by the FDR controlling procedures, with strong control on the Family Wise Error Rates. RESULTS: When there was no correlation between 1000 simulated features, the performance of the BH procedure was similar to the three proposed procedures. In low to medium correlation structures the BY procedure is too conservative. The BH procedure is too liberal, and the mean number of screened features was constant at the different levels of the correlation between features. The mean number of screened features by proposed procedures was between BY and BH procedures and reduced when the correlations increased. Where the features are highly correlated the number of screened features by proposed procedures reached the Bonferroni (BF) procedure, as expected. In real data analysis the BY, BH, M1, M2, and M3 procedures were done to screen gene expressions of colorectal cancer. To fit a predictive model based on the screened features the Efficient Bayesian Logistic Regression (EBLR) model was used. The fitted EBLR models based on the screened features by M1 and M2 procedures have minimum entropies and are more efficient than BY and BH procedures. CONCLUSION: The modified proposed procedures based on information theory, are much more flexible than BH and BY procedures for the amount of correlation between test statistics. The modified procedures avoided screening the non-informative features and so the number of screened features reduced with the increase in the level of correlation.

Robust and consistent measures of pattern separation based on information theory and demonstrated in the dentate gyrus.

Pattern separation is a valuable computational function performed by neuronal circuits, such as the dentate gyrus, where dissimilarity between inputs is increased, reducing noise and increasing the storage capacity of downstream networks. Pattern separation is studied from both in vivo experimental and computational perspectives and, a number of different measures (such as orthogonalisation, decorrelation, or spike train distance) have been applied to quantify the process of pattern separation. However, these are known to give conclusions that can differ qualitatively depending on the choice of measure and the parameters used to calculate it. We here demonstrate that arbitrarily increasing sparsity, a noticeable feature of dentate granule cell firing and one that is believed to be key to pattern separation, typically leads to improved classical measures for pattern separation even, inappropriately, up to the point where almost all information about the inputs is lost. Standard measures therefore both cannot differentiate between pattern separation and pattern destruction, and give results that may depend on arbitrary parameter choices. We propose that techniques from information theory, in particular mutual information, transfer entropy, and redundancy, should be applied to penalise the potential for lost information (often due to increased sparsity) that is neglected by existing measures. We compare five commonly-used measures of pattern separation with three novel techniques based on information theory, showing that the latter can be applied in a principled way and provide a robust and reliable measure for comparing the pattern separation performance of different neurons and networks. We demonstrate our new measures on detailed compartmental models of individual dentate granule cells and a dentate microcircuit, and show how structural changes associated with epilepsy affect pattern separation performance. We also demonstrate how our measures of pattern separation can predict pattern completion accuracy. Overall, our measures solve a widely acknowledged problem in assessing the pattern separation of neural circuits such as the dentate gyrus, as well as the cerebellum and mushroom body. Finally we provide a publicly available toolbox allowing for easy analysis of pattern separation in spike train ensembles.

Entropy removal of medical diagnostics.

Shannon entropy is a core concept in machine learning and information theory, particularly in decision tree modeling. To date, no studies have extensively and quantitatively applied Shannon entropy in a systematic way to quantify the entropy of clinical situations using diagnostic variables (true and false positives and negatives, respectively). Decision tree representations of medical decision-making tools can be generated using diagnostic variables found in literature and entropy removal can be calculated for these tools. This concept of clinical entropy removal has significant potential for further use to bring forth healthcare innovation, such as quantifying the impact of clinical guidelines and value of care and applications to Emergency Medicine scenarios where diagnostic accuracy in a limited time window is paramount. This analysis was done for 623 diagnostic tools and provided unique insights into their utility. For studies that provided detailed data on medical decision-making algorithms, bootstrapped datasets were generated from source data to perform comprehensive machine learning analysis on these algorithms and their constituent steps, which revealed a novel and thorough evaluation of medical diagnostic algorithms.

The Information Theory of Aging.

Information storage and retrieval is essential for all life. In biology, information is primarily stored in two distinct ways: the genome, comprising nucleic acids, acts as a foundational blueprint and the epigenome, consisting of chemical modifications to DNA and histone proteins, regulates gene expression patterns and endows cells with specific identities and functions. Unlike the stable, digital nature of genetic information, epigenetic information is stored in a digital-analog format, susceptible to alterations induced by diverse environmental signals and cellular damage. The Information Theory of Aging (ITOA) states that the aging process is driven by the progressive loss of youthful epigenetic information, the retrieval of which via epigenetic reprogramming can improve the function of damaged and aged tissues by catalyzing age reversal.

On distance-based topological indices and co-indices of fractal-type molecular graphs and their respective graph entropies.

In graph theory, a topological index is a numerical value that is in good correlation with certain physical properties of a molecule. It serves as an indicator of how a chemical structure behaves. The Shannon's entropy describes a comparable loss of data in information transmission networks. It has found use in the field of information theory. Inspired by the concept of Shannon's entropy, we have calculated some topological descriptors for fractal and Cayley-type dendrimer trees. We also find the entropy that is predicted by these indices.

Information theory-based direct causality measure to assess cardiac fibrillation dynamics.

Understanding the mechanism sustaining cardiac fibrillation can facilitate the personalization of treatment. Granger causality analysis can be used to determine the existence of a hierarchical fibrillation mechanism that is more amenable to ablation treatment in cardiac time-series data. Conventional Granger causality based on linear predictability may fail if the assumption is not met or given sparsely sampled, high-dimensional data. More recently developed information theory-based causality measures could potentially provide a more accurate estimate of the nonlinear coupling. However, despite their successful application to linear and nonlinear physical systems, their use is not known in the clinical field. Partial mutual information from mixed embedding (PMIME) was implemented to identify the direct coupling of cardiac electrophysiology signals. We show that PMIME requires less data and is more robust to extrinsic confounding factors. The algorithms were then extended for efficient characterization of fibrillation organization and hierarchy using clinical high-dimensional data. We show that PMIME network measures correlate well with the spatio-temporal organization of fibrillation and demonstrated that hierarchical type of fibrillation and drivers could be identified in a subset of ventricular fibrillation patients, such that regions of high hierarchy are associated with high dominant frequency.

Integrated information theory (IIT) 4.0: Formulating the properties of phenomenal existence in physical terms.

This paper presents Integrated Information Theory (IIT) 4.0. IIT aims to account for the properties of experience in physical (operational) terms. It identifies the essential properties of experience (axioms), infers the necessary and sufficient properties that its substrate must satisfy (postulates), and expresses them in mathematical terms. In principle, the postulates can be applied to any system of units in a state to determine whether it is conscious, to what degree, and in what way. IIT offers a parsimonious explanation of empirical evidence, makes testable predictions concerning both the presence and the quality of experience, and permits inferences and extrapolations. IIT 4.0 incorporates several developments of the past ten years, including a more accurate formulation of the axioms as postulates and mathematical expressions, the introduction of a unique measure of intrinsic information that is consistent with the postulates, and an explicit assessment of causal relations. By fully unfolding a system's irreducible cause-effect power, the distinctions and relations specified by a substrate can account for the quality of experience.

Phi fluctuates with surprisal: An empirical pre-study for the synthesis of the free energy principle and integrated information theory.

The Free Energy Principle (FEP) and Integrated Information Theory (IIT) are two ambitious theoretical approaches. The first aims to make a formal framework for describing self-organizing and life-like systems in general, and the second attempts a mathematical theory of conscious experience based on the intrinsic properties of a system. They are each concerned with complementary aspects of the properties of systems, one with life and behavior, the other with meaning and experience, so combining them has potential for scientific value. In this paper, we take a first step towards such a synthesis by expanding on the results of an earlier published evolutionary simulation study, which show a relationship between IIT-measures and fitness in differing complexities of tasks. We relate a basic information theoretic measure from the FEP, surprisal, to this result, finding that the surprisal of simulated agents' observations is inversely related to the general increase in fitness and integration over evolutionary time. Moreover, surprisal fluctuates together with IIT-based consciousness measures in within-trial time. This suggests that the consciousness measures used in IIT indirectly depend on the relation between the agent and the external world, and that it should therefore be possible to relate them to the theoretical concepts used in the FEP. Lastly, we suggest a future approach for investigating this relationship empirically.

Information theory-guided heuristic progressive multi-view coding.

Multi-view representation learning aims to capture comprehensive information from multiple views of a shared context. Recent works intuitively apply contrastive learning to different views in a pairwise manner, which is still scalable: view-specific noise is not filtered in learning view-shared representations; the fake negative pairs, where the negative terms are actually within the same class as the positive, and the real negative pairs are coequally treated; evenly measuring the similarities between terms might interfere with optimization. Importantly, few works study the theoretical framework of generalized self-supervised multi-view learning, especially for more than two views. To this end, we rethink the existing multi-view learning paradigm from the perspective of information theory and then propose a novel information theoretical framework for generalized multi-view learning. Guided by it, we build a multi-view coding method with a three-tier progressive architecture, namely Information theory-guided heuristic Progressive Multi-view Coding (IPMC). In the distribution-tier, IPMC aligns the distribution between views to reduce view-specific noise. In the set-tier, IPMC constructs self-adjusted contrasting pools, which are adaptively modified by a view filter. Lastly, in the instance-tier, we adopt a designed unified loss to learn representations and reduce the gradient interference. Theoretically and empirically, we demonstrate the superiority of IPMC over state-of-the-art methods.

Design of optimal labeling patterns for optical genome mapping via information theory.

MOTIVATION: Optical genome mapping (OGM) is a technique that extracts partial genomic information from optically imaged and linearized DNA fragments containing fluorescently labeled short sequence patterns. This information can be used for various genomic analyses and applications, such as the detection of structural variations and copy-number variations, epigenomic profiling, and microbial species identification. Currently, the choice of labeled patterns is based on the available biochemical methods and is not necessarily optimized for the application. RESULTS: In this work, we develop a model of OGM based on information theory, which enables the design of optimal labeling patterns for specific applications and target organism genomes. We validated the model through experimental OGM on human DNA and simulations on bacterial DNA. Our model predicts up to 10-fold improved accuracy by optimal choice of labeling patterns, which may guide future development of OGM biochemical labeling methods and significantly improve its accuracy and yield for applications such as epigenomic profiling and cultivation-free pathogen identification in clinical samples. AVAILABILITY AND IMPLEMENTATION: https://github.com/yevgenin/PatternCode.

Digital finance and performance in innovation: Evidence from universities.

This study evaluated 72 universities' performance innovation during 2011 to 2019 of panel data, using the data envelopment analysis-Malmquist method. The study used benchmark regression to analyse the relationship between digital finance and the universities' innovation performance. The aim was to improve innovation performance and promote national innovation across countries. According to the results of the empirical analysis, digital finance positively affects innovation performance. That finding was confirmed through advanced robustness test evaluation, such as limited information maximum likelihood, two-stage least squares, and interactive fixed effects. Moreover, based on information theory, the digital finance influence mechanism improves credit demand and financial efficiency. Additionally, innovation performance survived spatial overflow effects. Lastly, the paper concludes with some implications for improving digital financial coverage and constructing innovation networks among universities.

Deployment of entropy information theory in the Indian Sundarban region using hydrogeochemical parameters and GIS for assessment of irrigation suitability.

The evaluation of irrigation suitability plays a crucial role for the socio-economic development of the society, especially in the region of Sundarban. For sustainable agricultural practices, groundwater quality must be suitable for irrigation; otherwise, it can degrade soil and diminish crop yield. The entropy information theory, several irrigational indices, multivariate statistics, GIS, and geostatistics are used in this work to evaluate the geographical distribution and quality of groundwater in the Indian Sundarban region. In total, 33 groundwater samples were collected in 2018 (April and May), and they were evaluated for major cations, anions, as well as other parameters like electrical conductivity (EC), soluble sodium percentage (SSP), potential salinity (PS), total dissolved solids (TDS), Kelly ratio (KR), sodium absorption ratio (SAR), permeability index (PI), residual sodium carbonate (RSC), magnesium hazard (MH), and residual sodium bicarbonate (RSBC). The overall trend of the principal cations and anions is in the sequence of Na+ ≥ Mg2+ ≥ Ca2+ ≥ K2+ and HCO3- ≥ Cl- ≥ NO3- ≥ SO42- ≥ F-, respectively, whereas the spatial variation of %Na, SAR, RSBC, and MH demonstrate very poor irrigation water quality, and spatial variation of KR, RSC, SSP, PI, and PS signifies that the irrigation water quality is excellent to good. In order to identify the specific association and potential source of the dissolved chemical in the groundwater, statistical techniques like correlation and principal component analysis were also employed. The hydrochemical facies indicates that mixed type makes up the bulk (51.51%) of the water samples. Following the Wilcox plot, more than 75% of the water samples are good to doubtful; however, by the US salinity hazard map, roughly 60.60% of the samples had high salinity (C3-S1 zone). The EWQII reports that no samples fall into the very good (no restriction) category, whereas 30.30%, 30.30%, and 39.40% of the sample wells record good (low restriction), average (moderate restriction), and poor (severe restriction) irrigation water quality, respectively. Based on this study, the bulk of the groundwater samples taken from the study area are unsuitable for cultivation. The findings of this study will also help decision-makers develop adequate future plans for irrigation and groundwater resource management.